Neural networks are computing architectures modeled after the architecture of the human brain. As shown in FIG. 1, generally, a neural network 12 receives a plurality of inputs I.sub.0 -I.sub.m, operates on the inputs I.sub.0 -I.sub.m based upon a desired function, and produces a plurality of network outputs O.sub.0 -O.sub.p. Neural networks 12 are extremely fast because they process the inputs I.sub.0 -I.sub.m in parallel, in contrast to a computer which essentially runs in series. In structure, the neural network 12 comprises a plurality of "neurons" 14 which are organized in a series of successive layers. The layer nearest the outputs O.sub.0 -O.sub.p has neurons which are referred to as "output layer neurons," whereas the layer or layers situated prior to the output layer neurons have neurons which are referred to as "hidden layer neurons."
Each neuron 14 generally comprises the architecture as illustrated in FIG. 2. Any combination of available inputs may be provided to each neuron 14. In each neuron 14, a weight is initially applied to each input I which is connected to the neuron 14. As an example, FIG. 2 shows inputs I.sub.0 -I.sub.k passing into the neuron 14 in order to provide an output O.sub.q. Weights W.sub.0 -W.sub.k are applied respectively to the inputs I.sub.0 -I.sub.k by weight application mechanisms 16, which may simply be multipliers for mathematically combining the weight with a respective input I. After the weights W.sub.0 -W.sub.k have been applied to the respective inputs I.sub.0 -I.sub.k, the signals are mathematically combined via a summation operation, as indicated by the addition block 18 in FIG. 2. Next, the signal from the addition block 18 is sent to a nonlinear circuit 22, sometimes referred to in the art as a "sigmoid," where the signal undergoes a nonlinear transformation so as to provide the output O.sub.q.
Typically, there are numerous weight application mechanisms 16 as well as numerous weight values which can be applied by the weight application mechanisms 16 in each network 12. Accordingly, it is extremely difficult to determine the weights to be applied to a particular set of inputs in order to achieve a desired functionality for the network 12 and a desired set of outputs, because of the requisite complex mathematics involved. For this reason, much effort has been dedicated to creating learning algorithms, or training algorithms, for determining appropriate weights to be applied in order to achieve a certain network functionality.
In general, a learning algorithm compares a series of actual network outputs O.sub.0 -O.sub.p with a corresponding series of desired values O.sub.0 '-O.sub.p '. The series of desired values O.sub.0 '-O.sub.p ' are based upon a desired functionality of the network 12. With reference to FIG. 1, the comparisons generate a series of errors E.sub.0 -E.sub.p, as shown at error blocks 24. Further, the errors E.sub.0 -E.sub.p are usually mathematically made positive in such a way so as to insure that minimum error indicates the best match of the network functionality to the desired functionality. This procedure can be achieved by mathematical squaring, as indicated by dual arrows leading to multiplication blocks 25. Then, the squared values are mathematically combined via summation at an addition block 26 so as to provide a series of overall network errors E.sub.N. The series of network errors E.sub.N is then sent to a weight adjustment mechanism 28, which generates from time to time directions (either +D or -D for each weight) in which each weight change should be implemented in the neurons 14. As illustrated in FIGS. 1 and 2, the directions (+D or -D) are sent via many parallel connections 29 to magnitude adjustment mechanisms 31, which are sort of gain controls, and the magnitude adjustment mechanisms 31 determine the magnitude .delta. of the weight change and combine this magnitude with the sign of the appropriate direction. Thus, an incremental change of either +.delta. or -.delta. is provided to each of the weight application mechanisms 16.
A well known technique which is employed within the weight adjustment mechanism 28 of FIG. 1 is the back propagation learning algorithm. Entire textbooks have been written on this subject. In accordance with the back propagation learning algorithm, the gradient (slope on a graph) of an error is determined with respect to each weight, and the gradient indicates the direction in which each weight must be manipulated. In mathematics, this technique is sometimes referred to as a form of gradient descent.
However, most learning algorithms, including back propagation, require complex multiplication, summation, and generation of mathematical derivatives. These mathematical operations require enormous processing support, and much space must be utilized in order to implement multipliers and summers, especially when large numbers of neurons 14 are utilized. The serial operation of digital circuits can be used to reduce the size, but this results in slow processing time and a long learning period. If parallel analog circuits are utilized instead of digital circuits, learning speed is faster and the requisite area is reduced; however, non-idealities of analog circuits make it very difficult to achieve a learning algorithm exactly as expected, causing degradation of learning ability or sometimes even the absence of learning. The back propagation learning algorithm is particularly susceptible to these non-idealities.
Thus, a heretofore unaddressed need exists in the industry for a learning algorithm which may be employed in connection with a neural network to improve both signal processing speed and learning period without the need for complex circuitry, such as multiplication.